Let's look at this from two angles: flexibility and scalability.

First, flexibility. With AI becoming more capable, I feel like going from off-the-shelf software to something built for our own needs is getting easier and easier. Of course, getting some code running is only the beginning. We still have to figure out whether it actually works, and whether we want to maintain it.

We already have datafusion-distributed exploring this direction. I guess this kind of project could eventually let people replace parts of their Spark or Trino pipelines with something more tailored to their workload. How much of a pipeline? Well... that depends on what it needs.

So this got me thinking: could we also build a distributed version of DuckDB?

I'm not saying DataFusion or dfd is worse. I'm interested in what we could build with DuckDB and its extensions as the building blocks. How much can we reuse? How much do we have to add ourselves? And can we keep it convenient to use from an application once there are several machines involved?

Then there's scalability. Spark already gives us a lot: scheduling, resilience, distributed query planning, and a broad set of features. That's a pretty high bar. If we build our own distributed layer, those responsibilities don't just disappear.

But I think it's worth revisiting the tradeoff between scalability and system complexity. How far can we get with one machine first? When would adding a few workers actually help? And how much of that gain would we lose moving data around and coordinating everything?

So in this series, let's explore it. We'll use DuckDB and its extensions to build toward a distributed engine, with the coordinator inside the application and workers doing parts of the query remotely. I want to see whether we can make something flexible and easy to use, while still getting useful scalability and performance.

Will it be easier to use than Spark? Will it be fast enough to justify the extra machines? Let's build it and see. Though, well... recovery alone is going to give us plenty to think about.